The present disclosure relates to providing search results. Search engines typically operate by downloading third-party webpages and indexing their content within one or more search indexes. Some search engines populate the collection of webpages they index using one or more web crawlers that discover new webpages by following hyperlinks embedded within indexed webpages. Although web crawling may be effective for discovering a large number of webpages that might otherwise be difficult to locate within the world wide web, in some cases, web crawling may not be effective for indexing webpages in a timely manner.
For example, if a content provider publishes a new webpage containing news about a recent event, that webpage might not be indexed until days after its creation, if a search engine must rely solely on web crawling to discover its existence. As a result, the search engine may be limited in its ability to provide search results associated with webpages that were only recently created or modified prior to a client's search request. Moreover, even for webpages that have been indexed, it may be difficult for users to follow minute-by-minute developments with respect to a topic if content has been duplicated across multiple webpages.
Accordingly, search engines may be improved by techniques for providing streaming, non-duplicative search results to client devices in an automatic manner as new or modified relevant webpages are added to one or more search indexes.